Infusing Finetuning with Semantic Dependencies

Abstract For natural language processing systems, two kinds of evidence support the use text representations from neural models “pretrained” on large unannotated corpora: performance application-inspired benchmarks (Peters et al., 2018, inter alia), and emergence syntactic abstractions in those (Tenney 2019, alia). On other hand, lack grounded supervision calls into question how well these can ever capture meaning (Bender Koller, 2020). We apply novel probes to recent models— specifically focusing predicate-argument structure as operationalized by semantic dependencies (Ivanova 2012)—and find that, unlike syntax, semantics is not brought surface today’s pretrained models. then convolutional graph encoders explicitly incorporate parses task-specific finetuning, yielding benefits understanding (NLU) tasks GLUE benchmark. This approach demonstrates potential for general-purpose (rather than task-specific) linguistic supervision, above beyond conventional pretraining finetuning. Several diagnostics help localize our approach.1